{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Design space\n",
    "\n",
    "Currently, the following types of parameters are supported:\n",
    "\n",
    "- Integeral parameters\n",
    "- Continuous parameters\n",
    "- Continuous parameters, varying in log space (for example, we may want to search learing rate with in (1e-4, 1e-2))\n",
    "- Boolean parameters\n",
    "- Categorical parameters\n",
    "\n",
    "These built-in parameter types are managed by `DesignSpace` class, we'll firstly import the class\n",
    "\n",
    "## Define design space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright (C) 2020. Huawei Technologies Co., Ltd. All rights reserved.\n",
    "\n",
    "# This program is free software; you can redistribute it and/or modify it under\n",
    "# the terms of the MIT license.\n",
    "\n",
    "# This program is distributed in the hope that it will be useful, but WITHOUT ANY\n",
    "# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A\n",
    "# PARTICULAR PURPOSE. See the MIT License for more details.\n",
    "\n",
    "import torch\n",
    "from hebo.design_space.design_space import DesignSpace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want to optimize the hyper-parameters of a neural network, the hyper-parameters to be optimized are:\n",
    "\n",
    "- Size of hidden units `hidden_size`, it should be integer, the range is [16, 128]\n",
    "- Batch size `batch_size`, the range is also [16, 128]\n",
    "- Learning rate `lr`, the range is [1e-4, 1e-2], but we want it to vary in log space\n",
    "- Whether or not to use batch normalization `use_bn`, it should be a boolean parameter\n",
    "- Dropout rate `dropout_rate`, it's a continuous parameter, ranging from 0.1 to 0.5\n",
    "- Activation function `activation`, we define it as a categorical parameter, possible candidates are `relu`, `tanh` and `sigmoid`\n",
    "- Optimizer `optimizer`, it is also a categorical parameter, candidates are `sgd`, `adam` and `rmsprop`\n",
    "\n",
    "We can define a list of dictionary to specify the above hyper-parameters, and then pass the list to `DesignSpace` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = [\n",
    "    {'name' : 'hidden_size', 'type' : 'int', 'lb' : 16, 'ub' : 128},\n",
    "    {'name' : 'batch_size',  'type' : 'int', 'lb' : 16, 'ub' : 128},\n",
    "    {'name' : 'lr', 'type' : 'pow', 'lb' : 1e-4, 'ub' : 1e-2, 'base' : 10},  \n",
    "    {'name' : 'use_bn', 'type' : 'bool'},\n",
    "    {'name' : 'activation', 'type' : 'cat', 'categories' : ['relu', 'tanh','sigmoid']},\n",
    "    {'name' : 'dropout_rate', 'type' : 'num', 'lb' : 0.1, 'ub' : 0.9},\n",
    "    {'name' : 'optimizer', 'type' : 'cat', 'categories' : ['sgd', 'adam', 'rmsprop']}\n",
    "]\n",
    "\n",
    "space = DesignSpace().parse(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it, we have defined the search space, now we can do some random sampling. Runing `DesignSpace.sample()` would return a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hidden_size</th>\n",
       "      <th>batch_size</th>\n",
       "      <th>lr</th>\n",
       "      <th>use_bn</th>\n",
       "      <th>activation</th>\n",
       "      <th>dropout_rate</th>\n",
       "      <th>optimizer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27</td>\n",
       "      <td>107</td>\n",
       "      <td>0.000930</td>\n",
       "      <td>True</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.730755</td>\n",
       "      <td>rmsprop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>115</td>\n",
       "      <td>101</td>\n",
       "      <td>0.000348</td>\n",
       "      <td>False</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.115669</td>\n",
       "      <td>sgd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>27</td>\n",
       "      <td>91</td>\n",
       "      <td>0.000214</td>\n",
       "      <td>False</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.214109</td>\n",
       "      <td>adam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70</td>\n",
       "      <td>56</td>\n",
       "      <td>0.000851</td>\n",
       "      <td>True</td>\n",
       "      <td>sigmoid</td>\n",
       "      <td>0.763984</td>\n",
       "      <td>adam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>94</td>\n",
       "      <td>43</td>\n",
       "      <td>0.000815</td>\n",
       "      <td>True</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.615758</td>\n",
       "      <td>rmsprop</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   hidden_size  batch_size        lr  use_bn activation  dropout_rate  \\\n",
       "0           27         107  0.000930    True       tanh      0.730755   \n",
       "1          115         101  0.000348   False       tanh      0.115669   \n",
       "2           27          91  0.000214   False       tanh      0.214109   \n",
       "3           70          56  0.000851    True    sigmoid      0.763984   \n",
       "4           94          43  0.000815    True       tanh      0.615758   \n",
       "\n",
       "  optimizer  \n",
       "0   rmsprop  \n",
       "1       sgd  \n",
       "2      adam  \n",
       "3      adam  \n",
       "4   rmsprop  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inside `DesignSpace`: parameter transformation\n",
    "\n",
    "**NOTE**: You can skip this section if you don't need to define new parameter types or develop new BO algorithms.\n",
    "\n",
    "We can see that `DesignSpace.sample()` returns a pandas dataframe, that's how design parameters are represented, however, there are some drawbacks directly using the above dataframe to fit the surrogate model in BO:\n",
    "\n",
    "1. Categorical parameters are represented by  `str `, they should be transformed to integers\n",
    "2. For parameters varying in log space, it would be better to perform log transformation before feeding them to BO algorithms\n",
    "\n",
    "The `DesignSpace.transform` does the above two things, it transforms transforms categorical variables to integers and performs log transformation to parameters varying in log space. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DesignSpace.transform` takes a pandas dataframe as input, and returns a `torch.FloatTensor` and a `torch.LongTensor`: numerical and boolean parameters are transformed to `FloatTensor` (for boolean parameters, we can view `True/False` as 0/1), and categorical parameters would be transformed to a `LongTensor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hidden_size</th>\n",
       "      <th>batch_size</th>\n",
       "      <th>lr</th>\n",
       "      <th>use_bn</th>\n",
       "      <th>activation</th>\n",
       "      <th>dropout_rate</th>\n",
       "      <th>optimizer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>82</td>\n",
       "      <td>72</td>\n",
       "      <td>0.004158</td>\n",
       "      <td>False</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.896806</td>\n",
       "      <td>rmsprop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>99</td>\n",
       "      <td>0.000997</td>\n",
       "      <td>True</td>\n",
       "      <td>sigmoid</td>\n",
       "      <td>0.239654</td>\n",
       "      <td>sgd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>119</td>\n",
       "      <td>25</td>\n",
       "      <td>0.002203</td>\n",
       "      <td>False</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.186145</td>\n",
       "      <td>sgd</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   hidden_size  batch_size        lr  use_bn activation  dropout_rate  \\\n",
       "0           82          72  0.004158   False       tanh      0.896806   \n",
       "1           61          99  0.000997    True    sigmoid      0.239654   \n",
       "2          119          25  0.002203   False       tanh      0.186145   \n",
       "\n",
       "  optimizer  \n",
       "0   rmsprop  \n",
       "1       sgd  \n",
       "2       sgd  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "samp = space.sample(3)\n",
    "samp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([3, 5]), torch.Size([3, 2]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,xe = space.transform(samp)\n",
    "\n",
    "assert isinstance(x, torch.FloatTensor)\n",
    "assert isinstance(xe, torch.LongTensor)\n",
    "x.shape, xe.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The five numerical parameters are transformed to `x`, and the two categorical parameters are transformed to `xe`, the order of each column can be seen from `DesignSpace.numeric_names` and `DesignSpace.enum_names`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 82.0000,  72.0000,  -2.3811,   0.0000,   0.8968],\n",
       "         [ 61.0000,  99.0000,  -3.0012,   1.0000,   0.2397],\n",
       "         [119.0000,  25.0000,  -2.6570,   0.0000,   0.1861]]),\n",
       " ['hidden_size', 'batch_size', 'lr', 'use_bn', 'dropout_rate'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, space.numeric_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above cell, we can see that log transformation is performed to `lr`, and values of the boolean parameter `use_bn` is transformed to 0/1. \n",
    "\n",
    "The two categorical parameters are transformed to integers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[2, 1],\n",
       "         [1, 2],\n",
       "         [2, 2]]),\n",
       " ['activation', 'optimizer'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xe, space.enum_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `DesignSpace.inverse_transform` to recover the original dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hidden_size</th>\n",
       "      <th>batch_size</th>\n",
       "      <th>lr</th>\n",
       "      <th>use_bn</th>\n",
       "      <th>activation</th>\n",
       "      <th>dropout_rate</th>\n",
       "      <th>optimizer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>82</td>\n",
       "      <td>72</td>\n",
       "      <td>0.004158</td>\n",
       "      <td>False</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.896806</td>\n",
       "      <td>rmsprop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>99</td>\n",
       "      <td>0.000997</td>\n",
       "      <td>True</td>\n",
       "      <td>sigmoid</td>\n",
       "      <td>0.239654</td>\n",
       "      <td>sgd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>119</td>\n",
       "      <td>25</td>\n",
       "      <td>0.002203</td>\n",
       "      <td>False</td>\n",
       "      <td>tanh</td>\n",
       "      <td>0.186145</td>\n",
       "      <td>sgd</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   hidden_size  batch_size        lr  use_bn activation  dropout_rate  \\\n",
       "0           82          72  0.004158   False       tanh      0.896806   \n",
       "1           61          99  0.000997    True    sigmoid      0.239654   \n",
       "2          119          25  0.002203   False       tanh      0.186145   \n",
       "\n",
       "  optimizer  \n",
       "0   rmsprop  \n",
       "1       sgd  \n",
       "2       sgd  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space.inverse_transform(x,xe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bound of transformed parameters\n",
    "\n",
    "In `DesignSpace` class, the bound of the transformed parameters is automatically calculated, we can see the lower bound and upper bound using `DesignSapce.opt_lb` and `DesignSpace.opt_ub`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([16.0000, 16.0000, -4.0000,  0.0000,  0.1000,  0.0000,  0.0000],\n",
       "       dtype=torch.float64)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space.opt_lb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([128.0000, 128.0000,  -2.0000,   1.0000,   0.9000,   2.0000,   2.0000],\n",
       "       dtype=torch.float64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space.opt_ub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The order of bound vector elements is `space.numeric_names + space.enum_names`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hidden_size',\n",
       " 'batch_size',\n",
       " 'lr',\n",
       " 'use_bn',\n",
       " 'dropout_rate',\n",
       " 'activation',\n",
       " 'optimizer']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space.numeric_names + space.enum_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the third element of the bound vector is `lr`, and the range is transformed from [$10^{-4}$, $10^{-2}$] to [-4, -2]."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
